Skip to content

fix: fail over unhealthy Daytona sandbox hosts - #183

Merged
iamjr15 merged 1 commit into
mainfrom
fix/daytona-host-recovery-retries
Aug 8, 2026
Merged

fix: fail over unhealthy Daytona sandbox hosts#183
iamjr15 merged 1 commit into
mainfrom
fix/daytona-host-recovery-retries

Conversation

@iamjr15

@iamjr15 iamjr15 commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Detect Daytona's explicit host-recovery start rejection at the provider boundary.
  • Replace only the stopped canonical runtime while retaining the user's mounted workspace-volume subpath.
  • Extend durable preparation retries so remaining transient provider failures do not become immediate failed runs.

Architecture

A run still resolves the single canonical user sandbox first. If Daytona reports that the sandbox's assigned host is recovering, the existing runtime-replacement fence verifies the active-run lease and canonical volume identity, removes the disposable container, and creates one replacement attached to the same isolated volume subpath. Cloudflare Workflow retries remain the durable fallback around preparation.

Decisions Made

Decision Choice Alternatives considered Reasoning
Host-local recovery Replace the disposable runtime Wait indefinitely; fail the run The workspace is on a separate persistent volume, so failover preserves user files and avoids dependence on one unhealthy runner.
Failure classification Match Daytona's explicit 503 host-recovery rejection Replace on every 503 A narrow provider-boundary classifier prevents destructive failover during platform-wide outages.
Remaining transient failures Six exponential-backoff preparation retries Tight polling inside the Worker Durable retries do not hold an isolate or hammer Daytona.

Edge Cases Handled

Scenario Handling
Another run owns the sandbox Existing lease fence refuses replacement.
Sandbox identity or volume mount is ambiguous Existing canonical-runtime assertions fail closed.
Daytona changes or omits the host-recovery message No forced replacement; durable preparation retries apply.
Replacement creation is temporarily unavailable The Workflow retries the idempotent preparation step.

Verification

  • pnpm lint
  • pnpm typecheck
  • pnpm turbo build --force
  • pnpm deadcode
  • pnpm architecture:check
  • pnpm turbo skills:build
  • Resume the failed production Pomodoro run after deployment and verify the replacement sandbox preserves the project and serves the generated preview.

Preserve the mounted workspace volume while replacing a stopped runtime after
Daytona's explicit host-recovery rejection. Extend durable retries for other
transient preparation failures.
@vercel

vercel Bot commented Aug 8, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
cheatcode Skipped Skipped Aug 8, 2026 1:53pm

@iamjr15
iamjr15 merged commit 7ca2b51 into main Aug 8, 2026
6 checks passed
@iamjr15
iamjr15 deleted the fix/daytona-host-recovery-retries branch August 8, 2026 13:54
iamjr15 added a commit that referenced this pull request Aug 8, 2026
## Why

Daytona currently reports an unhealthy sandbox host with a structured
HTTP 503 body but no provider error code. The sandbox lifecycle should
not depend on provider message text.

## What changed

- validate the Daytona error projection at the adapter boundary
- normalize the specific host-recovery response to an internal
`daytona_host_recovering` code
- make lifecycle classification depend only on the internal code and
HTTP status
- document the normalized adapter-to-lifecycle contract

This preserves the guarded same-volume runtime replacement behavior
merged in #183 while keeping provider response details isolated to the
Daytona adapter.

## Verification

- `pnpm lint`
- `pnpm typecheck`
- `pnpm turbo build --force`
- `pnpm deadcode`
- `pnpm architecture:check`
- `pnpm turbo skills:build`
- captured the live Daytona 503 response shape for the affected sandbox
without exposing credentials

Production behavior will be exercised after merge and exact-SHA
Cloudflare deployment.
iamjr15 added a commit that referenced this pull request Aug 8, 2026
## Summary

- Treat provider `length` finish reasons as nonterminal model turns.
- Checkpoint the partial response and continue durably with an internal
instruction to split interrupted operations into smaller complete tool
calls.
- Validate the SDK finish-reason contract at the agent boundary and
accept only `stop` as tool-free semantic completion.
- Fail blocked, provider-error, and invalid terminal reasons explicitly
instead of publishing incomplete work as successful.

## Root cause

The production Pomodoro retry recovered its Daytona runtime correctly,
but Claude reached its per-response output limit while forming the next
operation. The Workflow used only the absence of parsed tool calls as
its completion condition, so it finalized that truncated turn and
exposed the sandbox readiness page.

## Architecture

The Mastra adapter validates the six AI SDK finish reasons. Cloudflare
Workflow owns the policy: tool calls execute durably; `length` appends a
non-visible continuation message and starts another checkpointed model
turn; only `stop` completes a tool-free run. This keeps semantic
completion in charge without adding a step, token, duration, or cost
ceiling.

## Decisions

| Decision | Choice | Reason |
|---|---|---|
| Handle truncation in Workflow | Durable continuation | It preserves
replay safety and applies to every provider and run type. |
| Preserve partial response history | Append an internal user
continuation message | The next turn sees the exact work already
attempted and avoids repeating completed actions. |
| Keep provider output settings unchanged | No fixed output-token
override | A larger fixed cap only postpones truncation and does not
establish correct completion semantics. |
| Reject non-stop terminal reasons | Explicit upstream failure | Content
filtering and provider errors are not successful outcomes. |

## Production evidence

- Run `019fe1b3-af54-7a69-bd1d-0cd940842e41` completed after model turn
5 returned `finishReason: "length"` with no tool call.
- The replacement Daytona sandbox was healthy and mounted the correct
durable volume, isolating this from the sandbox recovery issue fixed in
#183 and #184.

## Verification

- `pnpm lint`
- `pnpm typecheck`
- `pnpm turbo build --force`
- `pnpm deadcode`
- `pnpm architecture:check`
- `pnpm turbo skills:build`
- production flow will be repeated after merge and exact-SHA Cloudflare
deployment

## How to review

1. Review the finish-reason schema in
`packages/agent-core/src/mastra/durable-agent-step.ts`.
2. Review the durable loop policy in
`apps/agent-worker/src/durable-objects/agent-run-workflow.ts`.
3. Confirm the shared schema is enforced by
`agent-run-workflow-runtime.ts` and the README contract matches the
implementation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant